{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression: L1 vs L2\n",
    "-------------------------------\n",
    "\n",
    "This function shows how to use TensorFlow to solve linear regression via the matrix inverse.\n",
    "\n",
    "It is important to know the effect of loss functions in algorithm convergence. Here we will illustrate how the L1 and L2 loss functions affect convergence in linear regression.  We will use the same iris data set as in the prior recipe, but we will change our loss functions and learning rates to see how convergence changes.\n",
    "\n",
    "<img src=\"../images/04_L1_L2_learningrates.png\" width=\"512\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by loading the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from sklearn import datasets\n",
    "from tensorflow.python.framework import ops\n",
    "ops.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# L1-Loss\n",
    "--------------\n",
    "\n",
    "Here, we will illustrate linear regression with the L1-Loss. Later in this script, we will illustrate the same problem with L2-Loss.\n",
    "\n",
    "The equation for the L1 Loss for Linear Least Squares is\n",
    "\n",
    "$$S = \\sum_{i=1}^{N}\\left| y_{i} - \\hat{y_{i}} \\right|$$\n",
    "\n",
    "Where $N$ is the number of data points, $y_{i}$ is the i-th actual y-values, $\\hat{y_{i}}$ is the predicted i-th y-value.\n",
    "\n",
    "We start a computational graph session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we load the Iris data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iris.data = [(Sepal Length, Sepal Width, Petal Length, Petal Width)]\n",
    "iris = datasets.load_iris()\n",
    "x_vals = np.array([x[3] for x in iris.data])\n",
    "y_vals = np.array([y[0] for y in iris.data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set some model parameters.\n",
    "\n",
    "An important parameter to take note of is the learning rate.  If the learning rate is too large, the model will not converge.  If the learning rate is too small the model will converge too slowly.\n",
    "\n",
    "Here are two learning rate values to show convergence and non-convergence.\n",
    "\n",
    "Convergence happens below 0.35, try setting the learning rate less than that for convergence.  To illustrate non-convergence, set the learning rate to 0.4 or higher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 25\n",
    "learning_rate = 0.4 # Will not converge with learning rate at 0.4\n",
    "iterations = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can initialize placeholders, model variables, and model operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize placeholders\n",
    "x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)\n",
    "y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)\n",
    "\n",
    "# Create variables for linear regression\n",
    "A = tf.Variable(tf.random_normal(shape=[1,1]))\n",
    "b = tf.Variable(tf.random_normal(shape=[1,1]))\n",
    "\n",
    "# Declare model operations\n",
    "model_output = tf.add(tf.matmul(x_data, A), b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we declare the l1-loss function and the optimization function.  After that we initialize the model variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Declare loss functions\n",
    "loss_l1 = tf.reduce_mean(tf.abs(y_target - model_output))\n",
    "\n",
    "# Declare optimizers\n",
    "my_opt_l1 = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "train_step_l1 = my_opt_l1.minimize(loss_l1)\n",
    "\n",
    "# Initialize variables\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we start the training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #25 A = [[1.7771142]] b = [[3.5515566]]\n",
      "Step #50 A = [[0.79791415]] b = [[4.7515574]]\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "loss_vec_l1 = []\n",
    "for i in range(iterations):\n",
    "    rand_index = np.random.choice(len(x_vals), size=batch_size)\n",
    "    rand_x = np.transpose([x_vals[rand_index]])\n",
    "    rand_y = np.transpose([y_vals[rand_index]])\n",
    "    sess.run(train_step_l1, feed_dict={x_data: rand_x, y_target: rand_y})\n",
    "    temp_loss_l1 = sess.run(loss_l1, feed_dict={x_data: rand_x, y_target: rand_y})\n",
    "    loss_vec_l1.append(temp_loss_l1)\n",
    "    if (i+1)%25==0:\n",
    "        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)) + ' b = ' + str(sess.run(b)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# L2-Loss\n",
    "--------\n",
    "\n",
    "Here, we will illustrate linear regression with the L2-Loss..\n",
    "\n",
    "The equation for the L2 Loss for Linear Least Squares is\n",
    "\n",
    "$$S = \\sum_{i=1}^{N}\\left( y_{i} - \\hat{y_{i}} \\right)^{2}$$\n",
    "\n",
    "Where $N$ is the number of data points, $y_{i}$ is the i-th actual y-values, $\\hat{y_{i}}$ is the predicted i-th y-value.\n",
    "\n",
    "We start a computational graph session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# L2 Loss\n",
    "# Reinitialize graph\n",
    "ops.reset_default_graph()\n",
    "\n",
    "# Create graph\n",
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same as before, we initialize the placeholders, variables, and model operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize placeholders\n",
    "x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)\n",
    "y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)\n",
    "\n",
    "# Create variables for linear regression\n",
    "A = tf.Variable(tf.random_normal(shape=[1,1]))\n",
    "b = tf.Variable(tf.random_normal(shape=[1,1]))\n",
    "\n",
    "# Declare model operations\n",
    "model_output = tf.add(tf.matmul(x_data, A), b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the loss function, variable initialization, and optimization functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Declare loss functions\n",
    "loss_l2 = tf.reduce_mean(tf.square(y_target - model_output))\n",
    "\n",
    "# Declare optimizers\n",
    "my_opt_l2 = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "train_step_l2 = my_opt_l2.minimize(loss_l2)\n",
    "\n",
    "# Initialize variables\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start our linear regression training with the L2 function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #25 A = [[109.81761]] b = [[81.18325]]\n",
      "Step #50 A = [[-3926.9023]] b = [[-6179.8516]]\n"
     ]
    }
   ],
   "source": [
    "loss_vec_l2 = []\n",
    "for i in range(iterations):\n",
    "    rand_index = np.random.choice(len(x_vals), size=batch_size)\n",
    "    rand_x = np.transpose([x_vals[rand_index]])\n",
    "    rand_y = np.transpose([y_vals[rand_index]])\n",
    "    sess.run(train_step_l2, feed_dict={x_data: rand_x, y_target: rand_y})\n",
    "    temp_loss_l2 = sess.run(loss_l2, feed_dict={x_data: rand_x, y_target: rand_y})\n",
    "    loss_vec_l2.append(temp_loss_l2)\n",
    "    if (i+1)%25==0:\n",
    "        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)) + ' b = ' + str(sess.run(b)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is matplotlib code to plot the loss for the L1 and L2 loss functions applied to the same linear regression problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl81NW9//HXJyEkJCDI5gJisFqUIqJQdytiF7VeUate61Jc7nVp3WqrtdpW5VYtrXq1PtpatVbrvtR9uda6lp+4gAtKUVEkyCJL2GSH5PP743wHJsnMZAL5zkxm3s/HYx6Z+W7n851MPjlzvud7jrk7IiJS/MryHYCIiOSGEr6ISIlQwhcRKRFK+CIiJUIJX0SkRCjhi4iUCCV8aVdmdoqZjc93HJI/ZvasmY3JdxzSkhJ+B2ZmM8zsmymWdzazh6P1bmYj8xBeC2ZWG8XTKcW6MWY2ycyWmdksM/ttqu2Stncz2zHeiPPDzEaY2VNmttjMlpjZv83sKjPbMt+xNWdmV5jZ3cnL3P1Qd78zXzFJekr4xWs8cBLwRb4DyVI1cAHQG9gLOBj4aV4jilmaf3z7Ai8D/w/Y2d17AIcA64Hd8h2fdHDurkcHfQAzgG+2ss0sYGQr25wKTAW+BKYDZyatGxkd4yfAfGAucGrS+l7AE8Ay4E3gf4DxacqpBRzolMW5XQg8mWG9AzumWF4G/AKoi+L9G9A9WlcF3A3UA0uAt4CtonWnROf+JfAZcGKacq8AHgYeiLZ9G9gtaf22wN+BBdFxzkux793R+/VfKY4/Hrgpi/fntOh3thh4Dti+2XtzFjAtOs8/ANaGfX8U7ftZtOxG4PMo5knAAdHyQ4C1wDpgOfBetPzlxLm18vtIfB7GADOBhcBl+f67KuZH3gPQYzN+ee2X8L8LfAUw4EBgJbBHtG4koXY5FqgADovWbxmtvx94EKgBhgCzaZ+E/xjwmwzr0yX804BPgB2ArsAjwF3RujOBJwnfJsqB4cAWUezLgEHRdtsAX0tT7hVRgjsmej9+SkjsFVFymwT8CugcxTAd+E6zfY+Mtu3S7Ng1QEMWv6/R0TnuAnSKEuprzd6bp4AewADCP59D2rDv80DPRHyEb4q9ou1/QvjWWJV0Tnc3i+9lNib8TL+PxOfhVqAL4RvMGmCXfP9tFesj7wG0CAhuJ9QEPshi2wHAS8A7wGTgsHzHn+P3agbtkPBT7PMYcH70fCSwiqQkHf1+9o6S5jpC00Ni3dVsZsKPksQsoHeGbdIl/BeAHya9HhTF2Ck67mvA0Gb71BBqwt9rnoRTHP8K4PWk12WEbz0HEJqiZjbb/ufAX5P2fTXDsftH55X8fv42im0F8Ito2bPA6c1iWElUU4+OsX/S+geBS9qw76hW3oPFRN9qaD3hZ/p9JD4P/ZPWvwkcn+u/pVJ5FGIb/h2Er4rZ+AXwoLvvDhwP/DGuoIqZmR1qZq+b2SIzW0KoxfdO2qTe3dcnvV5JqK31Ifzhfp60rm4zYzkSuAY41N0XbsIhtm0WQx0hxq2AuwhNGPeb2ZzownCFu68A/pPQDDLXzJ42s50zlLHhfN29kfDPaVtge2Db6ELrkui9vDQqu8W+KSwGGgnfMBLHv9hDO/6j0XkQlXNjUhmLCN/O+iUdK/naTeL3le2+TWI0s5+a2VQzWxrt052mn49MMv0+WotV2lnBJXx3f5XwIdzAzL5iZv8X9eL4V9IfoxO+kkP4EM7JYahFwcwqCW3O1xLas3sAzxCSQGsWEJp7tktaNmAzYjmE8PX+P9z9/U08zBxCUkuOZz0wz93XufuV7j4Y2Bc4HPgBgLs/5+7fIiTbD6M40tlwvmZWRqiZzyEkys/cvUfSo5u7H5a0b9rhaaN/PG8AR7dyjp8TrrMkl9PF3V9rZb9s990Qo5kdAFwMHEdoxusBLGXj56O14XbT/j6yiFXaWcEl/DRuAc519+GENtNETf4K4CQzm0VIUufmJ7y8qjCzqqRHJwiJ3Myqom06R+tSJfHOQCVR8jazQ4FvZ1OwuzcQ2mSvMLNqMxtMuADXmspmMZeZ2SjgHuB77v5mNuWz8bwSj3LgPuDHZjbQzLoSmpgecPf1ZnaQme0abbeM0LTQaGZbmdloM6shtCEvJ9S00xluZkdH7/UF0T6vE5ojvjSzn5lZFzMrN7MhZvb1LM8HQnI9zcwuMbO+AGbWHxiYtM3NwM/N7GvR+u5mdmyWx2/rvt0ICXoB0MnMfsXGShaExF0b/eNLJe3vI8t4pR0VfMKPPiT7Ag+Z2bvAn9n4lff7wB3u3p/QDHFXhg9esXqG0MaeeFwRLf8oet2P0IyxiqY1LQDc/UvgPEI772LgBEKvm2ydQ/gK/gWhOe6vWeyzvFnMo4BfEr6lPWNmy6PHs60cZ0qz45xKuAZ0F/Aq4WLqajZWBLYm9JJZRuil8kq0bRmhV9AcwrfLA4GzM5T7OKEJaDFwMnB09O2hgfCtYVhU9kLgtui8suLu4wnvxzeAj6MmlP8jtIvfFG3zKDCO0DS1DPgAODTL47d13+ei8j8mNMespmmTz0PRz3ozezvF/pl+H5Jj5l54E6CYWS3wlLsPMbMtgI/cfZsU200h9D74PHo9Hdjb3efnMl4pHWZ2BeFi8Un5jkWkrQq+Nuzuy4DPEl87LUjcgDKTcIMOZrYLoZ/1grwEKiJS4Aou4ZvZfcAEYFB0i/3pwInA6Wb2HuFr/Oho858A/x0tvw84xQvxK4uISAEoyCYdERFpfwVXwxcRkXgU1OBIvXv39tra2nyHISLSYUyaNGmhu/fJZtuCSvi1tbVMnDgx32GIiHQYZpb13e1q0hERKRFK+CIiJUIJX0SkRBRUG76IlK5169Yxa9YsVq9ene9QClJVVRX9+/enoqJik4+hhC8iBWHWrFl069aN2tpaUo/zV7rcnfr6embNmsXAgQNb3yENNemISEFYvXo1vXr1UrJPwczo1avXZn/7UcIXkYKhZJ9ee7w3SvgiIiVCCV9EJNK1a8vZFV999VX22GMPOnXqxMMPP9ymfZtoaID1+Z33RQlfRCSDAQMGcMcdd3DCCSds3oE++gg++aR9gtpESvgiIhnU1tYydOhQysrani5nzJjBqFGjGDp0KAePGcPMKOE/9NBDDBkyhN12241vfOMbAEyZMoU999yTYcOGMXToUKZNm9au5wHqlikiBeiCCy7g3XffbddjDhs2jBtuuKFdj9mac889lzFjxjDm5JO5/YorOO/aa3ns8MMZO3Yszz33HP369WPJkiUA3HzzzZx//vmceOKJrF27loaGhnaPRzV8EZGYTJgwITQFNTRw8mGHMT76J7bffvtxyimncOutt25I7Pvssw9XX30148aNo66uji5durR7PKrhi0jByXVNPHbNaus333wzb7zxBk8//TTDhw9n0qRJnHDCCey11148/fTTHHbYYfz5z39m1KhR7RqGavgiIjHZd999uf/++6GhgXuefZYDvv51AD799FP22msvxo4dS58+ffj888+ZPn06O+ywA+eddx6jR49m8uTJ7R6PavgiIpGVK1fSv3//Da8vvPBCDjjgAI466igWL17Mk08+yeWXX86UKVOy2vemm27i1FNP5Xe//S19unfnr7ffDsBFF13EtGnTcHcOPvhgdtttN8aNG8ddd91FRUUFW2+9NZdeemm7n19BzWk7YsQI1wQoIqVp6tSp7LLLLvkOIz4NDbBuHVRWwibeNZvqPTKzSe4+Ipv91aQjIhK39eth7lz44IMW7fm5pIQvIhK3hQvhiy/C88bGvIWhhC8iErfkWr0SvohIEUseQ0cJX0SkiBVIDV/dMkVE4tbQABUVsM020Llz3sJQDV9EJJJqiOPrr7+ewYMHhwHQDj6Yurq6rPfdoHdv2G476Nu3uBO+mZWb2Ttm9lTcZYmItLfdd9+diRMnMnnyZI455hguvvjith9kyy2hRw9YuTL0xc+TXNTwzwem5qAcEZF2d9BBB1FdXQ3A3nvvzaxZs7Led8PwyEOGcPCoUcx88UVYsqQ4h0c2s/7Ad4GrgAvjLEtEiszIkS2XHXcc/PCHoaZ82GEt159ySngsXAjHHNN03csvb3ZIf/nLXzj00EOz3n7D8MhDhnD7Cy+E4ZEfeKBoh0e+AbgYyN9laRGRdnD33XczceJELrrootDN8tNPW52ycMKECZxw/PHgzsnHHReGR25sLL7hkc3scGC+u08ys5EZtjsDOAPCVGIiIkDmGnl1deb1vXu3S40+4Z///CdXXXUVr7zyCpWVlbB0KSxeDFttBdnMZQuhlw5AY2NRDo+8H3CEmc0A7gdGmdndzTdy91vcfYS7j+jTp0+M4YiItN0777zDmWeeyRNPPEHfvn3DwkTNvrw847777rsv9993HwD3PPooB+y+OzQ2Ft/wyO7+c+DnAFEN/6fuflJc5YmIbK5UQxw/88wzLF++nGOPPRYILRFP3Hpr2GDZMoiaXtIOj/yDH/C7OXPos+22/PWmm6BnTy46+eTiHR45KeEfnmk7DY8sUro61PDIc+aER9++0FpT9Lp1oQloiy02uw/+5g6PnJM7bd39ZeDlXJQlIhK7RLt8KxdtgdB237t3eL5iRRgLP+rmmWu601ZEpK169Ag/s0n4a9bA8uXgDjNnQhv68bc3JXwRKRiFNANfRt26Qffu2SX8RYvgww9Dwi8r2+TB09rjvVHCF5GCUFVVRX19fcdI+qtWheTdKYtW8YaG0IxTVrbJCd/dqa+vp6qqahOC3UijZYpIQejfvz+zZs1iwYIF+Q6ldbNnhwuwffrA1FZGjqmvD/8gpk6FBQvCRdxNmNO2qqqqSS+gTaGELyIFoaKigoEDB+Y7jOx84xtw7LHwxz+2vu1xx8H774eEP25cuCFsxoy4I0xJTToiIm3R2Bja5SdNgv33D90zM1myJIyWCXDBBXD77fHHmIZq+CIibbFsWUj6VVXw6qswbx5su2367ceOhbVrw/Nhw3ITYxqq4YuItEV9ffj51a+GnwsXZt5+771DExDAtGnw6KOhx04eKOGLiLRF797wt7/B0UeH161dZH7yydAtE+Dvfw/7rV4db4xpKOGLiLRF9+5w8smw557hdaYavntI8HfdFV7X1ISfK1bEG2MaSvgiIm0xezaMHx+GRxg+PPwDSGfFinBzVuLO3MSQCitXxh9nCrpoKyLSFo89BuecEy7WtjbYYzSb1YaEn6jh5ynhq4YvItIWiYu2ia6WmTRP+Hmu4Svhi4i0xaJFYajjigo44ww44YT02yYSfuKfw377ha6ciR4+OaYmHRGRtqivh549w/OFC+Hjj9NvO3RoSPBDhoTXvXrBAQfEH2MaquGLiLTFokUhcUMYSydTL50ttggJPlHDX7o09NiZPj3+OFNQwhcRaYsrr4TrrgvPe/cOCT/djVSTJ8Pdd4cB0wDmz4cf/ABeey03sTajhC8i0hYjRsCBB4bnvXuH4Y+XLk297eOPhz77CbpoKyLSgTz22MY7ZwcPhsMP3zhWTnOLF4eumBUV4bW6ZYqIdBCNjfC978E994TX3/lOGDqhb9/U2y9ZsrFLJmys4etOWxGRArd0aUj6iV46rWme8Dt3DrNkqYYvIlLgFi0KPxO9dObMgX79wmBqqSSPhZ/w1ltw3nnxxZiB+uGLiGQrkfATNfzu3UPSnzs39fZ33AFr1jRdlscx8ZXwRUSy1TzhV1eHiVDS9cUfMKDlsvvuC/8oDjssnhgzUJOOiEi29t47jJS5667htVnmm69+/3uYMKHpsmuugdtuizfONJTwRUSy1b17GA+nW7eNy3r3Tj0JSmNjmMP22WebLq+uzlsvHTXpiIhk6403wjSFJ520cdmRR0JlZcttv/wy3IGb3EsHQsLXePgiIgXu3nvDhdjkhP+rX6XetvnQyAk1NeFCbx6oSUdEJFvJA6cla2houSxdws9jDV8JX0QkW8lDIyf87nfhhqrmwyssXhx+Nu+Hf+ON8MIL8cWYgZp0RESylaqG37VruEBbXw/bbLNx+X77weeft9x+663jjzMN1fBFRLKVqobfp0/42bynTkUF9O8PXbo0Xf7qq3D11fHFmIESvohItv75Txg3rumy3r3Dz+Z98V99Ff7nf1o29bz4Ilx2WfhWkGNK+CIi2dp++5Z3z6ZL+C+8AJdfHgZLS5YYInnVqnhizEAJX0QkG8uXh9r9Bx80Xd6vH5x7Lgwc2HT5kiVhisOyZmk2j0Mk66KtiEg2vvgCLrkEtt1246TkEHrh/P73LbdvPjRyQh4nQYmthm9mVWb2ppm9Z2ZTzOzKuMoSEYld84HTkq1ZA8uWNV2WamhkyOs0h3E26awBRrn7bsAw4BAz2zvG8kRE4pMp4e+6K5x5ZtNl6Wr4RxwRjjVoUPvH2IrYmnTc3YHl0cuK6JFmancRkQJXXx9+pkr4qQZQe+GF1Bdmq6rCIw9ivWhrZuVm9i4wH3je3d9Isc0ZZjbRzCYuSDXinIhIIWg+21Wy3r1b9tLp1KnpqJoJs2eHawFTprR/jK2INeG7e4O7DwP6A3ua2ZAU29zi7iPcfUSfxA0MIiKF5swzQ7JOVcNPNSb+j38MzzzTctvFi0Nvn6lT44kzg5x0y3T3JcBLwCG5KE9EpN117hx66DTvZgkbm3Q8arVevx5uuAEmTmy5bR4v2sbWhm9mfYB17r7EzLoA3wLGtbKbiEhhuusuWLoUzjmn5bpDDw01/4aG0JSzdGlYnuqibTEmfGAb4E4zKyd8k3jQ3Z+KsTwRkfjce29otkmV8EeODI+EdEMjQ3EmfHefDOwe1/FFRHJq0aLU7fcQxsuZPRv69g03ViUSfgn1wxcRKR7pJj8BePtt2GEHeOWV8Hr58jBaZqoafqdO4Uatyy6LL9Y0NLSCiEg2Ug2NnNB8ALUDDwxJPZ3Onds3tiyphi8i0prGxjB0Qroafqox8c3CI5Vf/SrMjZtjSvgiIq0pKws19ksvTb1+iy1CE06ihv/443DaaS3Hwk+4/374xz/iiTUDJXwRkWyUl0NlZep1Zk3vtn3zTfjb38I/gVRqavIyPLISvohIaz7+ONxp+9FH6bcZNw5OPDE8Twyclq5Jp7pavXRERArSJ5/ALbeEYRHSOfnkjX3x042UmVBdrRq+iEhByjRwWsLs2TBpUniebiz8hO7dUw/REDN1yxQRaU2msfATrroKHnwwtONXVoapD9N5+OH2jS9LSvgiIq1JJPxMzTR9+oTtGhrgkUdyE1cbqUlHRKQ169bBVluFnjrp9O4dRstM/HPI5N574dRT2y++LCnhi4i05qqrwiTmmSRuvlq4MExjeM896bd9//2Q9HOs1YRvZl8xs8ro+UgzO8/MMnyvEREpQYnhFWbNgiefhLq69NtWV4ebstavz01skWxq+H8HGsxsR+AWYDsg9/+aRETy5Sc/geuuy7zNbrvBAw/A1luH1611y4Sc98XPJuE3uvt64CjgJne/iDDWvYhIaXj88TAiZiZ9+sBxx20cGC1Twq+pCT9znPCz6aWzzsy+D4wB/iNaluZ+YRGRIpRppMwEd3jppY1NOZkS/pZbhukS0421E5NsEv6pwFnAVe7+mZkNBO6KNywRkQLR0BBupMp00xWEYRRGj4YhQ2DQoI0XcVP5z/8MjxxrNeG7+7+B8wDMbEugm7trbloRKQ2J2ataq+FDuHC7445h/tsClE0vnZfNbAsz6wm8DdxqZtfHH5qISAFYsQIGDoRtsrh02adP0zHx0/n3v+Hww+GddzY/vjbI5qJtd3dfBhwN/M3d9wK+GW9YIiIFYsAAmD4djj229W1794bnnguDqGVqn1+xAp5+Ooy/k0PZJPxOZrYNcBzwVMzxiIh0XIl2+wkT0o+FD3nrpZNNwh8LPAd86u5vmdkOwLR4wxIRKRDPPAPf/jbMm9f6thddBEOHZh4LHzb2w8/xEMnZXLR9CHgo6fV04HtxBiUiUjCmTYPnn89cY08YMgR23jnzBOZQuDdemVl/M3vUzOZHj7+bWf9cBCcikneLFoXaevfurW/7+edhiORMtXsITTo77bSxaSdHsumH/1fCUAqJKxYnRcu+FVdQIiIFo74+NNFkGikzYeLE8DPT5CcQEv3HH29+bG2UTRt+H3f/q7uvjx53ABnuKBARKSKLFrV+01VC4qLtlVfGF89myCbh15vZSWZWHj1OAurjDkxEpCBssw3ssUd22yZGzMymL/6RR8K11256XJsgm4R/GqFL5hfAXOAY4JQYYxIRKRzXXRdGwcxGIuFnU8N/++1wA1YOtZrw3b3O3Y9w9z7u3tfdj0S9dEREWurZM1yMzSbhV1fnvFvmps54dWG7RiEiUqj22QduvDG7bcvKwsXY449vfduamsLrlplGK32ORESKwPr18PrrsHhx+x87DzX8bLplpuLtGoWISCFKjJSZbS+dthg6NEyOnkNpE76ZfUnqxG5Al9giEhEpFPVRh8RshkZuqz/8of2P2Yq0Cd/du+UyEBGRgrNoUfgZRw0/Dza1DV9EpPhVVsKoUdA/htFkrr46DKOcQ5vahi8iUvz22ANeeCGeYy9YUJAToGwSM9vOzF4ys3+b2RQzOz+uskREOpzq6g7TLTMb64GfuPtgYG/gR2Y2OMbyRETa12WXwfDh8Ry7ujp0+8w0M1Y726SEb2bvt7aNu89197ej518CU4F+m1KeiEhefPghrFoVz7HzMCZ+pm6ZR6dbBWzdlkLMrBbYHXgjxbozgDMABgwY0JbDiojEa+bMMKdtHL7ylTCTVmNjPMdPIdNF2weAe0jdF78q2wLMrCvwd+CCaDL0Jtz9FuAWgBEjRuiGLhEpHHV12Y+U2VZHHBEeOZQp4U8GrnX3D5qvMLNvZnNwM6sgJPt73P2RTQtRRCQPVq4MPWm23z7fkbSbTG34FwAtauSRo1o7sJkZ8BdgqrtfvwmxiYjkz+rVMGYM7LlnPMcfPx5qa+Gtt+I5fgqZ7rT9V4b99gcmtnLs/YCTgffN7N1o2aXu/kzbQhQRyYOePeGOO+I7vntoMlq6NL4ymtnUG68uBG7ItIG7j0ejaopIR7V2LVRUtD4h+abKQy8dDY8sIpLK2LFh8vKGhniOX1MTfuZwiORNTfjqTSMixW3mzJDwy8vjOX6B9cPX8MgiUrrq6uLrgw/QvTscdVQ8A7OloeGRRURSqauD/feP7/jdu8Mjue2truGRRUSaW78eZs0qqj74oIQvItLS2rVwySVw8MHxllNbC5deGm8ZSTQevohIc9XV8Otfx1/OypXxTJCehmr4IiLNLVkSpjf0mDsk5nhMfCV8EZHm/vjHMI9t3MlYCV9EJM9mzgwJP3FzVFxqagqjH76ISMmqq8tND53Ro6FL7m5rUsIXEWmurg4GDYq/nF/8Iv4ykqhJR0QkmXto0slVH/y4LwwnUcIXEUnW2AjXXw/HHht/WaefHqY6zBE16YiIJCsvhzPOyE1ZFRUdYrRMEZHi9MUX8N57sG5d/GXluJeOEr6ISLIHHoBhw3IzE1V1dajh56gdXwlfRCTZzJmhq2SvXvGXVV0dkv2aNfGXhRK+iEhTiT74cU1tmGyvveDHP85ZDV8XbUVEkuXqpiuAUaPCI0dUwxcRSRb3TFfJGhth+fIw/n4OKOGLiCS7804466zclPXkk9CtG7z/fk6KU5OOiEiyQw/NXVmJicxz1BdfNXwRkYSZM0Ote/ny3JSXSPg56ouvhC8ikvCPf8ARR0B9fW7KU8IXEcmTujooK4N+/XJTXmK8fTXpiIjk2MyZIdl3ytHlzb594Ze/hK99LSfF6aKtiEhCLvvgA/ToAWPH5qw41fBFRBJynfDdYd68MGl6Dijhi4gkPPVUaGLJpW23hWuvzUlRatIREUnIUVv6Bmahp4566YiI5NCMGfCnP8GCBbktt6ZGvXRERHJqwgT44Q9zn/BVwxcRybG6uvAzlxdtIacJX234IiIQEn6vXhtvhsqViy6CLbfMSVGxJXwzux04HJjv7kPiKkdEpF3MnJm7YZGTjRmTs6LibNK5AzgkxuOLiLSfXPfBT5g7Fz75JCdFxVbDd/dXzaw2ruOLiLSr8eNh1arcl3v++WE8/KlTYy8q7xdtzewMM5toZhMX5PrquIhIQo8esM02uS+3lHrpuPst7j7C3Uf06dMn3+GISCmaMQMuvRSmT8992ZdfDs8+m5Oi8p7wRUTy7r334JprcjcOfrKBA2Hw4JwUpYQvIpKvPvg5FlvCN7P7gAnAIDObZWanx1WWiMhmqauDLl2gyJuV4+yl8/24ji0i0q7efBN23DEMZlbE1KQjIqVt0aJw09Upp+Q7kthpaAURKW09e8Knn8L69fmOJHZK+CJSupYvh4oKqKzM3Ty2eaQmHREpXb/7XegW+eWX+Y4kJ5TwRaQ0rV4dJjwZPhy6dct3NDmhhC8ipem++8JkJxdckO9IckYJX0RKjzvceCMMGQKjRuU7mpwp/qsUIiLNvf56GE7httuKvu99MiV8ESk9e+8NL74YfpYQJXwRKT1mcNBB+Y4i59SGLyKl5Ze/DPPIuuc7kpxTwheR0rFsWbhY+8UXJdV2n6CELyKl4/bbw01WJdQVM5na8EWk+LnDAw/AlVfCfvuFm61KkGr4IlL8Vq8OUxh+9atw5535jiZvVMMXkeLkDk88Ad/+dpjc5MUXoX//khgkLR3V8EWk+MybB8ccA0ceCbfcEpbV1pZ0sgfV8EWk2CxZAsOGhYlNfvMb+NGP8h1RwVDCF5Hicuedodvl+PHhAq1soCYdESkuDz4YhkxQsm9BNXwRKS7//CfMnZvvKAqSavgiUjzcQ4+cHXbIdyQFSQlfRIrDnDkweDC88kq+IylYSvgiUhxuuw0+/BD69ct3JAVLCV9EOr7160N/++98B3bcMd/RFCwlfBHp+J58EmbPhrPPznckBU0JX0Q6vj/9CbbbDr773XxHUtDULVNEOr6zzw4DpJX40Amt0bsjIh3fUUflO4IOQU06ItJxrVoFv/51GEpBWqWELyKFbfp0uPtu+PjjlvPQPvBAmKP2ww/zE1sHoyYdESlM69bBtdfC2LGhfR7gtdfZmPVqAAAMAElEQVRgn32grg7WrAkXa3fZBQ48ML+xdhBK+CJSeFauDIl98mQ4+mi4+GJ4772NUxPedBNcd114fuONJTkh+aZQwheRwrFuHVRUQHV16GI5diyMHh3W7bXXxu1++MMwXeG0aXDaafmJtQMyb94mlkcjRozwiRMn5jsMEcm1xYvhoYfCBdjHHoM99sh3RB2GmU1y9xHZbKsavojkx9q18MgjcN998OyzoXa/++5Qpr4kcYk14ZvZIcCNQDlwm7v/Js7yRKRANTaGeWbr6kJiP+CA0OPmrLOgpgbOOQe+/30YMULt8TGKLeGbWTnwB+BbwCzgLTN7wt3/HVeZIgWnsTHUZBsawvOGhvDo1g06dw79yOfPD4N/JT923DEkwtmz4e23Q4+U1avDMbp2hVGjoEeP0BRSXx/GgDfb2G1xq63CXadLl4ZtkstvbAzt3506hflfV6yAysqNj9WrYYstwnFeew0mTgz93JctC7Xv6uowVyyE2aUmTw5JPPHo2hWuuSasP/dceOYZ+PzzsA5g6NBwAbayEt54I5xreXlufy8lKs4a/p7AJ+4+HcDM7gdGA+2e8MeddBLHPfpoi+U377ADb/TqxS7LlnHxRx+1WH/DTjvxXo8e7L54Med98kmL9b/ZeWc+6taNferrOWP69BbrLx88mJk1NRw0fz4/qKtrsf6SXXdlXlUVh86dy3GzZrVYf8GwYSytqOCo2bM5Ys6cFuvP2mMP1pSXc8LMmXx73rwm6xw49etfB+DUzz7jwIULm6xfVV7O2VE76NmffsqeixYBkKg7Lamo4IJhwwC48OOPGbp0aZP951ZV8fNddwXg0qlT+ery5U3Wf1ZTw5WDBwMwdsoUBqxc2WT91G7dGLfzzgD8dvJktlqzBku6XvT2lltyw047AXDTO+/QPZEMIq/16sXNX/kKALdOmkRlQ0OT9S/27csdtbWYO3e+9RbNPbP11tw/YABd1q/nlrff3nDeiRge7t+fR/v1o+fatdz0zjtN9nUz7h4wgP/bemv6rVrFbydPxpLeO3Pnlh124MW+fdlx+XKuef99INzU0qmxkU7ujBs0iFf79GH44sVcN3lyi/h+NmQIb/Tqxf4LF/LrKVNarD93t914v0cPvjVvHpel6GN++vDhfNq1K6Nnz+bHKT67x++5J1906cIJM2dyxmeftVg/ep99WNq5M6d/9hknz5zZZF0j8K0DDqChrIzzp03jqDlzWG/GivJyyoDVZWUc+/rrAFw2dSqj5s+nwYz1ZWWsN2NhZSWnTZgAwJgZM9hu1Srmb7018yormV9VxZzOnakbObJFTKVs2LBh3HDDDbGXE2fC7wd8nvR6FrBX843M7AzgDIABAwZsUkHrOnWirrq6xfIV0bgaq8vKUq5fFdUqVpaXp1y/OmpLXJ5m/dpo/Zdpyl8XfTVdVlGRcn0ihS1Js96j/es7d+azmpqm65KeL6ysbLF+TVI76ILKyibHdzO+TBpz5IuqKrqtX99k//rOnTc8n9OlCxXNLu7Pqara8Hx2VRUNzb6Gf5G0fmZ1NcubjXEyN2n9ZzU1dG1W/ryk9dNraqhobAyxJ51TwrSuXWluYbS+0YyPunVrsi/Awuj81pkxZYstaN6IsLiiAgifgSlRbdfNNhwj8f4sLy/nzZ49N5S13ox1Zhven1ldunBrbS0NZjQmHrDh9zGta1d+M2gQDWYbHo3RewbwZs+enLnHHqwtK2OtGZhR1dDA5126ADBpyy25etAgKhsbN8TmZiyL4n+9Z08Wde5MYxRfovxV0e9jfO/ezKuspMKdzo2NVDQ2sq6sjHLC5/OO2lruqK1lWadOGz6Pya7aZReu2mWXFssT7qytTbtOci+2XjpmdgxwiLv/V/T6ZGAvdz8n3T7qpSMi0jZt6aUT5+Xw2cB2Sa/7R8tERCQP4kz4bwE7mdlAM+sMHA88EWN5IiKSQWxt+O6+3szOAZ4jdMu83d1bXp0SEZGciLUfvrs/AzwTZxkiIpId3dImIlIilPBFREqEEr6ISIlQwhcRKREFNTyymS0AWo5RkJ3ewMJWtyo+Ou/SovMuLdmc9/bu3iebgxVUwt8cZjYx27vNionOu7TovEtLe5+3mnREREqEEr6ISIkopoR/S74DyBOdd2nReZeWdj3vomnDFxGRzIqphi8iIhko4YuIlIgOn/DN7BAz+8jMPjGzS/IdT5zM7HYzm29mHyQt62lmz5vZtOjnlvmMsb2Z2XZm9pKZ/dvMppjZ+dHyoj5vADOrMrM3zey96NyvjJYPNLM3os/8A9Hw40XFzMrN7B0zeyp6XfTnDGBmM8zsfTN718wmRsva7bPeoRN+0kTphwKDge+b2eD8RhWrO4BDmi27BHjB3XcCXoheF5P1wE/cfTCwN/Cj6Hdc7OcNsAYY5e67AcOAQ8xsb2Ac8L/uviOwGDg9jzHG5XxgatLrUjjnhIPcfVhS//t2+6x36IRP0kTp7r4WSEyUXpTc/VVgUbPFo4E7o+d3AkfmNKiYuftcd387ev4lIQn0o8jPG8CDxOzxFdHDgVHAw9Hyojt3M+sPfBe4LXptFPk5t6LdPusdPeGnmii9X55iyZet3H1u9PwLYKt8BhMnM6sFdgfeoETOO2raeBeYDzwPfAoscffErO/F+Jm/AbgYaIxe96L4zznBgX+Y2SQzOyNa1m6f9VgnQJHccnc3s6LsZ2tmXYG/Axe4+7JQ6QuK+bzdvQEYZmY9gEeBnfMcUqzM7HBgvrtPMrOR+Y4nD/Z399lm1hd43sw+TF65uZ/1jl7D10TpMM/MtgGIfs7PczztzswqCMn+Hnd/JFpc9OedzN2XAC8B+wA9zCxRWSu2z/x+wBFmNoPQRDsKuJHiPucN3H129HM+4R/8nrTjZ72jJ3xNlB7Od0z0fAzweB5jaXdR++1fgKnufn3SqqI+bwAz6xPV7DGzLsC3CNcwXgKOiTYrqnN395+7e393ryX8Pb/o7idSxOecYGY1ZtYt8Rz4NvAB7fhZ7/B32prZYYQ2v8RE6VflOaTYmNl9wEjCkKnzgMuBx4AHgQGEoaWPc/fmF3Y7LDPbH/gX8D4b23QvJbTjF+15A5jZUMJFunJC5exBdx9rZjsQar89gXeAk9x9Tf4ijUfUpPNTdz+8FM45OsdHo5edgHvd/Soz60U7fdY7fMIXEZHsdPQmHRERyZISvohIiVDCFxEpEUr4IiIlQglfRKREKOFLh2ZmW5nZvWY2PbodfYKZHZWnWEaa2b5Jr88ysx/kIxaRVDS0gnRY0U1ZjwF3uvsJ0bLtgSNiLLNT0pguzY0ElgOvAbj7zXHFIbIp1A9fOiwzOxj4lbsfmGJdOfAbQhKuBP7g7n+Obua5AlgIDAEmEW7icTMbDlwPdI3Wn+Luc83sZeBdYH/gPuBj4BdAZ6AeOBHoArwONAALgHOBg4Hl7n6tmQ0DbgaqCQOgnebui6NjvwEcBPQATnf3f7XfuySykZp0pCP7GvB2mnWnA0vd/evA14H/NrOB0brdgQsIcyjsAOwXjddzE3CMuw8HbgeS79ru7O4j3P06YDywt7vvTrj782J3n0FI6P8bjWXePGn/DfiZuw8l3DV8edK6Tu6+ZxTT5YjERE06UjTM7A+EWvhawi3oQ80sMf5Kd2CnaN2b7j4r2uddoBZYQqjxPx+NxFkOzE06/ANJz/sDD0QDWXUGPmslru5AD3d/JVp0J/BQ0iaJAeEmRbGIxEIJXzqyKcD3Ei/c/Udm1huYCMwEznX355J3iJp0ksdgaSD8HRgwxd33SVPWiqTnNwHXu/sTSU1EmyMRTyIWkVioSUc6sheBKjM7O2lZdfTzOeDsqKkGM/tqNAJhOh8Bfcxsn2j7CjP7Wpptu7NxeN4xScu/BLo139jdlwKLzeyAaNHJwCvNtxOJm2oT0mFFF1qPBP7XzC4mXCxdAfyM0GRSC7wd9eZZQIap4dx9bdT88/uoCaYTYRTWKSk2vwJ4yMwWE/7pJK4NPAk8bGajCRdtk40BbjazamA6cGrbz1hk86iXjohIiVCTjohIiVDCFxEpEUr4IiIlQglfRKREKOGLiJQIJXwRkRKhhC8iUiL+PyfUsvGfOrg8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot loss over time\n",
    "plt.plot(loss_vec_l1, 'k-', label='L1 Loss')\n",
    "plt.plot(loss_vec_l2, 'r--', label='L2 Loss')\n",
    "plt.title('L1 and L2 Loss per Generation')\n",
    "plt.xlabel('Generation')\n",
    "plt.ylabel('L1 Loss')\n",
    "plt.legend(loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
